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TITLE OF THE INVENTION : : 

INFORMATION PROCESSING METHOD, INFORMATION PROCESSOR, AND 

CONTROL PROGRAM 

5 

PA CKGROUNP OF THE INVENTION 
Field of the Invention 

[0001] The present invention relates to an information 
10 processing technology for editing or playing back data. 

Description of the Relate d Art 

[0002] Enhanced performance and lower cost of compact 
computer systems have promoted widespread use of home 
electric appliances having a computer used for control 

15 thereof or information processing. The uses of home video 

equipment have been changed from analog recording of 
broadcasted information and playback of video and music data 
supplied through media to recording of video and audio data 
in a form of digital data having high quality and durability. 

20 Also, compact and inexpensive video cameras which can be 

purchased by ordinary households have become common, and it 
is not uncommon sight that video recording is performed in 
the home so as to enjoy watching the recorded data. 
[0003] Further, since computers and the Internet, which 

25 is a global network, have been widely used in homes, high- 
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quality contents supplied in a form of digital data, such as 
video and audio data, can be dealt with more easily. 
Accordingly, multimedia data containing video, audio, and 
text data have been widely distributed. 
5 [0004] In addition, people have more chances to do 

creative activities, as can be seen by many personal sites 
on the Internet . 

[0005] Under these circumstances, users' need is more 
than recording data and watching supplied data, and their 
10 desire for performing video editing in the home has been 

growing, which has been conventionally performed by 
broadcasting companies or the like. 

[0006] When video data is edited in the home, the video 
data may be edited while copying the data from a playback 

15 device to a recording device, for example, from a VTR to 

another VTR, or from a video camera to a VTR. In this 
method, a desired scene is searched for by fast -forwarding 
or rewinding a master tape for playback, and the video data 
is edited while copying the data to a recording tape. In 

20 this method, by using two or more playback devices or by 

using a video editing device or a computer device so as to 
copy data to a recording device, a special editing effect 
can be added to the video data, for example, a special 
transition effect can be inserted between scenes or a telop 

2 5 and caption can be synthesized into the video data. However, 
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performing this technique requires experience in exclusive 
editing devices and editing, and also time and effort are 
needed. Therefore, it is difficult for amateur users to 
master this technique. 
5 [0007] Recently, however, video data has been able to be 

taken into a computer device or the like so as to edit the 
video data by using a video capture card, an IEEE 139 4 
interface, a DV editing card, or the like. In this method, 
various editing effects can be added by using commercially 

10 available video editing software. 

[0008] In particular, PCs having a good performance can 
be purchased in a reasonable price and PCs have become 
common in homes, and software having a professional editing 
function is available in the market. Therefore, editing 

15 methods using computer devices or the like have become 

mainstream. 

[0009] Also, some of recently- available digital video 
cameras have a simple video editing function of, for example, 
adding a simple transition effect and inputting a title. In 

20 this type of video camera, various editing effects can be 

added during or after recording. In a method of editing 
data while copying it by using this type of video camera as 
a playback device, an editing effect can be added to the 
data without using a video editing device, for example, 

25 unnecessary part can be deleted or scenes can be rearranged. 
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[0010] In the future, the price of video cameras having 
an editing function will be reduced and an editing function 
will advance, so that video cameras having an editing 
function will be pervasive. Accordingly, users who cannot 
5 operate computers will be able to edit video data, and thus 

a video editing function will become familiar to more users. 
[0011] With a growing demand for performing video editing 
in the home, an environment where video editing can be 
performed without a dedicated editing device is becoming a 

10 reality owing to a high-performance PC and video camera. 

[0012] However, the following problems occur in the 
above-described known techniques. That is, expert knowledge 
and techniques are required to edit multimedia data, 
particularly video data, and complicated operation has to be 

15 performed. Therefore, editing video data recorded by a home 

video camera is still difficult for users who are not 
familiar with video editing. 

[0013] As described above, an editing function of 
software for editing video data in a computer device and an 

20 editing function provided in a video camera have been 

improved so that amateur users can perform video editing 
relatively easily. However, users have to understand 
technical terms and have the know-how to edit data, and thus 
this type of software is not always easy to understand for 

25 beginners who don't have expert knowledge about video 



- 5 - 



editing. Additionally, the edited data is not always 
satisfactory for the user. 

[0014] Specifically, in an example of commercially 
available video editing software, a user can freely select 
5 and arrange scenes so as to connect the scenes , and a 

transition clip can be arbitrarily specified and inserted 
between scenes. Also, video cameras having an editing 
function of adding an arbitrary transition clip between 
scenes are available . 

10 [0015] In the above-described method of arbitrarily 

selecting a transition clip, however, if a user is not 
familiar with video editing and does not have expert 
knowledge about editing, the user may be confused not 
knowing which clip should be inserted or the user may select 

15 a clip which is unsuitable for the theme or situation of 

scenes, so that unnatural video with excessive editing 
effects may be created. 

[0016] Also, in another example of software for easily 
editing video data, templates of edit scenario in accordance 

20 with various themes (event information), such as a child's 

athletic festival, birthday party, and wedding ceremony, are 
prepared, and video data can be edited simply by taking 
recorded scenes from a video tape and arranging them. In 
this method, the user only has to arrange scenes in 

25 accordance with a specified order and no complicated 
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operation is needed, and thus video editing can be performed 
relatively easily even by a beginner, 
[0017] In this type of software, however, since 
situations and transition clips which can be inserted in 
5 each theme (event information) are specified by edit 

scenario, the edit freedom is restricted and user's 
personality cannot be expressed. Further, transition clips 
specified in edit templates do not always satisfy the 
preference and need of the user. 
10 [0018] In addition, as described above, a transition clip 

can be inserted between scenes not only when two scenes are 
connected so as to create a sequence of video data but also 
when two or more scenes are played back sequentially. In 
that case, too, the above-described problems occur. 

15 

SUMMARY OF THE INVENTION 

[0019] The present invention has been made in view of the 
above-described problems and it is an object of the present 

2 0 invention to provide a method in which a user who does not 

have expert knowledge about editing can easily edit video 
data by inserting a transition clip between scenes. 
[0020] It is another object of the present invention to 
provide a method in which a user who is not familiar with an 

25 editing technique can create sophisticated video including a 



video effect. 

[0021] The present invention provides an information 
processing method for editing input data, including an 
obtaining step of obtaining metadata of the data; a 
selecting step of selecting a transition clip used for 
adding a transition effect to the data based on the 
metadata; and a processing step of adding a transition 
effect to the data by using the transition clip. 
[0022] Further objects, features, and advantages of the 
present invention will become apparent from the following 
description of the preferred embodiments with reference to 
the attached drawings. 
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[0023] Fig. 1 is a block diagram showing the entire 
configuration of an information processing system including 
an information processor according to a first embodiment of 
the present invention. 

[0024] Fig. 2 shows a screen used for instructing to 
insert a transition clip in the information processor 
according to the first embodiment. 

[0025] Fig. 3 includes tables showing the relationship 
between video data and metadata attached thereto in the 
information processor according to the first embodiment. 
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[ 0026 ] Fig . 4 is a flowchart showing the entire process 
of inserting a transition clip in the information processor 
according to the first embodiment. 

[0027] Fig. 5 is a flowchart showing a process of 
5 extracting potential transition clips in the information 

processor according to the first embodiment. 
[0028] Fig. 6 is a flowchart showing a process of 
determining a transition clip in the information processor 
according to the first embodiment. 

10 [0029] Fig. 7 shows the relationship between event 

information of metadata and transition clips in the 
information processor according to the first embodiment. 
[0030] Fig. 8 shows information about each transition 
clip in the information processor according to the first 

1 5 embodiment . 

[0031] Fig. 9 shows the relationship between metadata and 
meanings of each transition clip in the information 
processor according to the first embodiment. 
[0032] Fig. 10 defines the correlation in metadata and 

20 features in the information processor according to the first 

embodiment . 

[0033] Fig. 11 is a flowchart showing the entire process 
of inserting a transition clip in an information processor 
according to a second embodiment . 
25 [0034] Fig. 12 is a flowchart showing a process of 
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extracting transition clips which are unsuitable for 
transition of two scenes in the information processor 
according to the second embodiment . 

[0035] Fig. 13 shows a screen for displaying an error 
5 message when an unsuitable transition clip is specified in 

the information processor according to the second embodiment. 
[0036] Fig. 14 shows a screen used for instructing to 
insert a transition clip in an information processor 
according to a third embodiment . 

10 [0037] Fig. 15 is a flowchart showing the entire process 

of inserting a transition clip so as to edit video data in 
the information processor according to the third embodiment. 
[0038] Fig. 16 is a flowchart showing a process of 
extracting potential transition clips in the information 

15 processor according to the third embodiment. 

[0039] Fig. 17 is a flowchart showing a process of 
extracting potential transition clips in the information 
processor according to the third embodiment. 

20 DESCRIPT ION OF THE PREFERRED EMBODIMENTS 

[0040] Hereinafter, embodiments of the present invention 
will be described with reference to the drawings. 
<First Embodiment > 
25 [0041] In this embodiment, video data input to a computer 
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device is edited so as to set a transition effect (technique 
used for connecting scenes) between scenes. 
[0042] In order to input video data recorded by a 
recording device such as a video camera into a computer 
5 device, for example, the video data stored in an external 

storage medium may be input to the computer device or may be 
input through a video capture card or an IEEE 1394 interface 
or the like. The input data may be stored in units of files, 
each including a clip (part or small unit of video) or a 

10 plurality of clips. 

[0043] A transition effect can be set by using metadata 
attached to video data. The metadata includes contents of 
multimedia data and is used in an application such as search, 
the contents can being described based on a schema 

15 standardized by MPEG- 7 or the like. 

[0044] Fig. 1 shows an example of the entire 
configuration of an information processing system including 
an information processor according to an embodiment of the 
present invention . 

20 [0045] In Fig. 1, the information processor includes a 

microprocessor (CPU) 11, which executes operations for 
various processes and logical decision and controls devices 
connected to an address bus AB, a control bus CB, and a data 
bus DB. The operations thereof are instructed by programs 

25 stored in a ROM 12 or a RAM 13, which will be described 



later. Also, a plurality of computer programs can be 
operated in parallel by using the function of the CPU 11 and 
the mechanism of the computer programs . 
[0046] The address bus AB is used for transferring 
address signals for indicating devices to be controlled by 
the CPU 11. The control bus CB is used for transferring and 
applying control signals for devices to be controlled by the 
CPU 11. The data bus DB is used for transferring data among 
the devices . 

[0047] The ROM 12 is a fixed read-only memory and stores 
control programs such as processing programs executed in 
this embodiment. Also, the ROM 12 stores a computer program 
area and a data area for storing the procedure of control 
executed by the CPU 11. 

[0048] The recordable RAM 13 is used as the computer 
program area and the data area for storing the procedure of 
control executed by the CPU 11, and is also used as a 
temporary storage area for various computer programs and 
various types of data transmitted from the devices other 
than the CPU 11. 

[0049] These storage media such as the ROM 12 and the RAM 
13 store computer programs and data used for realizing data 
editing of this embodiment. The data editing function is 
realized when the CPU 11 reads and executes program codes 
stored in these storage media, but the type of storage media 
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is not limited. 

[0050] A recording medium containing the programs and 
data according to the present invention may be supplied to 
the system or the apparatus so as to copy the programs and 
5 data from the recording medium to a rewritable medium such 

as the RAM 13. In that case, a CD-ROM # a floppy (registered 
trademark) disk, a hard disk, a memory card, or a 
magnetooptocal disk can be used as the recording medium. 
[0051] A hard disk (DISK) 14 functions as an external 

10 memory for storing various computer programs and data. The 

hard disk (DISK) 14 includes a storage medium which can 
read/write a large amount of information at a relatively 
high speed, and the various computer programs and data can 
be stored in and taken from the storage medium as necessary. 

15 The stored computer programs and data are entirely or 

partially invoked to the RAM 13 as required by a command 
input through a keyboard or instructions of the various 
computer programs . 

[0052] As the recording media for storing these programs 
20 and data, a ROM, a floppy (registered trademark) disk, a CD- 

ROM, a memory card, or a magnetooptical disk can be used. 
[0053] A memory card (MemCard) 15 is a removable storage 
medium. By storing information in this storage medium and 
connecting the storage medium to another device, the stored 
2 5 information can be referred to and transferred. 
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[0054] A keyboard (KB) 16 includes various function keys, 
such as alphabet keys , Hiragana keys , Katakana keys , 
punctuation/ symbol keys, and cursor-movement keys for moving 
a cursor. Also, a pointing device such as a mouse may be 
5 included therein. 

[0055] A cursor register (CR) 17 is also provided. 
Contents of the cursor register can be read/written by using 
the CPU 11. A CRT controller (CRTC) 19, which will be 
described later, displays a cursor in a display device (CRT) 
10 20 at a position corresponding to an address stored in the 

cursor register. 

[0056] A display buffer memory (DBUF) 18 stores patterns 
of data to be displayed. 

[0057] The CRT controller (CRTC) 19 displays contents 
15 accumulated in the display buffer (DBUF) 18 in the display 

device (CRT) 20. 

[0058] The display device (CRT) 20 includes a cathode ray 
tube, and the CRT controller 19 controls the display pattern 
of dot configuration and display of the cursor in the 
20 display device (CRT) 20. 

[0059] A character generator (CG) 21 stores patterns of 
characters and symbols to be displayed in the display device 
(CRT) 20. 

[0060] A communication device (NCU) 22 is used for 
25 communicating with another computer device or the like. By 
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using the communication device (NCU) 22, the programs and 
data of this embodiment can be shared with other devices . 
In Fig. 1, the NCU 22 is connected to a personal computer 
(PC), a device (TV/VR) for receiving, storing, and 
5 displaying video data of television broadcast or video data 

taken by a user, and a home game computer (GC) through a 
network (LAN), so that information can be freely exchanged 
among these devices. Of course, any device may be connected 
to the apparatus of the present invention through the 
10 network. Also, the type of network is not restricted to the 

closed network as shown in Fig. 1, and the network may be 
connected to the external network. 

[0061] A receiver (DTU) 23 realizes a receiving function 
of broadcast communication using an artificial satellite or 

15 the like, and has a function of receiving radio waves 

broadcasted through the artificial satellite by a parabolic 
antenna (ANT) and extracting broadcasted data. There are 
many types of broadcast communication: broadcast using 
ground waves; broadcast using coaxial cables and optical 

20 cables; broadcast using the LAN and a large-scale network, 

etc, but any type of broadcast communication may be adopted. 
[0062] In the information processing system including the 
above -described devices, by connecting an IEEE 1394 terminal 
of a video camera or the like to an IEEE 1394 terminal (DV 

25 terminal) supplied from the communication device (NCU) 22, 
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the video camera can be controlled by the computer device, 
video and audio data recorded in the video camera can be 
captured and taken into the computer device, and the data 
can be stored in the storage devices such as the ROM 12, the 
5 RAM 13, the hard disk (DISK) 14, and the memory card 

(MemCard) 15 shown in Fig, 1. Further, the data may be 
stored in another storage device through the LAN or the like 
so as to use the data . 

[0063] The present invention can be realized by supplying 

10 a recording medium containing programs according to the 

present invention to the system or the apparatus so that a 
computer in the system or the apparatus reads and executes 
program codes stored in the recording medium. 
[0064] Fig. 2 shows an example of a screen in which a 

15 user selects a desired clip from among a plurality of 

transition clips, the process being illustrated in Fig. 6. 
This is an example when a window system is used, and this 
screen is displayed in the display device (CRT) 20 by the 
information processor of this embodiment. 

20 [0065] In Fig. 2, reference numeral 21 denotes a title 

bar, which is used for operating the entire window, for 
example, for performing movement and changing the size. 
[0066] Reference numeral 22 denotes a list box for 
displaying a list of transition clips which are suitable for 

25 transition of scenes specified by an operator. The operator 
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can select a transition clip to be inserted. In Fig. 2, the 
list includes "open heart", "cross zoom", "cross fade", and 
so on, and "cross zoom" is currently selected and is 
highlighted. When the operator presses a cursor-movement 
key on the keyboard (KB) 16, the highlight is shifted from 
"cross zoom" to "open heart" or "cross fade". In this way, 
the operator can arbitrarily select a desired transition 
clip from the list. 

[0067] An area 23 is used for displaying an image of a 
highlighted transition clip. The operator can check an 
image of transition of scenes by seeing animation sample. 
[0068] An area 2 4 at the lower part of the screen is used 
for displaying the explanation of a highlighted transition 
clip in a text form. In Fig. 2, the currently-highlighted 
"cross zoom" is explained. 

[0069] In this embodiment, the image and explanation of a 
transition clip are displayed at the same time, so that the 
user can easily understand what each transition clip is like. 
Animation samples and text to be displayed in the areas 23 
and 24 are stored in a storage medium, such as the hard disk 
(DISK) 14 shown in Fig. 1. Alternatively, the samples and 
text may be stored in the PC on the LAN through the 
communication device (NCU) 22 or in a computer on the 
external network through the receiver (DTU) 23 shown in Fig. 
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[0070] Buttons 25 to 27 can be pressed by operating the 
mouse or keys in the keyboard (KB) 16. 

[0071] The button 25 is an detailed setting button, which 
is used by the operator to arbitrarily set detailed 
5 information such as the direction or length of a transition 

clip. A screen displayed when the detailed setting button 
is selected and detailed items which can be set are 
different depending on the type transition clip. 
[0072] The button 26 is an OK button, which is used for 

10 finally confirming a currently- selected transition clip and 

input detailed information. When the OK button is selected, 
the transition clip which is currently highlighted in the 
list box 22 and the detailed information which has been 
input by pressing the button 25 are confirmed, and then the 

15 process proceeds to store the confirmed information. 

[0073] The button 27 is a cancel button, which is used 
for canceling the input data. 

[0074] In order to set a transition effect by using the 
information processor according to the present invention, 
20 metadata attached to video data is used. The metadata can 

be described in accordance with a method standardized by, 
for example, MPEG- 7. 

[0075] Hereinafter, the metadata attached to the video 
data in the information processor according to the present 
25 invention will be described. 
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[0076] Fig. 3 shows an example of video data and metadata 
attached thereto. The figure shows that metadata is 
attached to a series of frames included in the video data. 
The metadata is information representing the contents and 
5 features of each piece of data, such as a type of event, 

characters (hereinafter, characters and objects related to 
an event will be referred to as "objects"), situation, and 
place. Herein, the contents and features of data are 
expressed by words (keywords) and character information 

10 (text) is mainly stored. However, free-form sentences, 

grammatically- analyzed sentences, and sentences including 
5W1H may be also described. Also, information about the 
relationship between event information and objects, the 
relationship between scenes , information having a 

15 hierarchical structure and relative importance, and 

nonverval information in which the feature of data is 
described in a way the computer can easily process the 
information, can be attached. 

[0077] Video data and the corresponding metadata are 
20 stored in a recording medium, such as the hard disk (DISK) 

14 in Fig. 1. Also, data stored in the PC on the LAN can be 
used through the communication device (NCU) 22 and data 
stored in a computer on the external network can be used 
through the receiver (DTU) 23. 
25 [0078] Now, a process of editing video data by using a 
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transition clip in the information processor according to 
the present invention will be described with reference to a 
specific example. 

[0079] Fig. 4 is a flowchart showing a process of 
5 inserting a transition clip when video data is edited. 

[0080] In step S41, specification of two scenes to be 
edited is accepted. The user can specify scenes and a 
transition clip by operating the keyboard (KB) 16 shown in 
Fig. 1 so as to specify each material (clip) and to place 

10 the clip on a time line or a story board by using video 

editing software or the like operated in the information 
processor of this embodiment. Further, a desired part can 
be extracted from a video clip by specifying the start and 
end points as necessary. 

15 [0081] Herein, a scene means a section in the video data 

to be edited which the user wants to adopt, and is a minimum 
unit in data editing. Information about an edited scene can 
be represented by, for example, a frame ID of start and end 
points of a section which is adopted in a video clip. 

20 [0082] The specified scenes are stored in a table holding 

a video editing status. The table includes information 
about video editing status, such as selected scenes, 
playback order of the scenes, and special effects including 
a telop and transition clip to be inserted into video. The 

25 table is stored in a recording medium such as the DISK 14 or 
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the RAM 13 in Fig. 1. 

[0083] In step S42, instructions to insert a transition 
clip between the scenes specified by the user are accepted. 
[0084] In this embodiment, arbitrary two scenes are 
5 selected first and then a transition clip is set between the 

two scenes. Alternatively, all scenes to be used may be 
selected so as to decide the playback order, and then 
transition clips used for transition of scenes may be set. 
[0085] In step S43, metadata corresponding to the two 

10 scenes sandwiching the point for a transition clip is 

obtained. An example of the metadata is shown in Fig. 3, 
which is stored in a recording medium such as the DISK 14 in 
Fig. 1. The obtained metadata is stored in a recording 
medium such as the RAM 13 in Fig. 1, and is used in step S44. 

15 [0086] In step S44, the metadata obtained in step S43 is 

checked so as to obtain potential transition clips which are 
suitable for transition of the scenes. Potential transition 
clips can be obtained by referring to a table shown in Fig. 
7, showing the relationship between event information of 

20 metadata attached to the two scenes and transition clips. 

For example, when event information in the metadata attached 
to a first scene is "reception-makeover" and when event 
information in the metadata attached to a second scene is 
"reception-candle service", transition clips such as open 

25 heart, cross fade, and slide are searched for. 
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[0087] Alternatively, the relationship in the metadata 
attached to the specified scenes may be analyzed, and a 
suitable transition clip may be searched for based on the 
analysis result and on the meaning and effects of transition 
5 clips. The process performed in that case will be described 

later with reference to a flowchart shown in Fig. 5. 
[0088] In step S45, it is determined whether or not a 
potential transition clip has been extracted in step S44. 
If a potential transition clip exists, the process proceeds 

10 to step S46, and otherwise, the process is completed. 

[0089] In step S46, it is determined whether or not a 
plurality of potential transition clips exist. If a 
plurality of potential transition clips exist, the process 
proceeds to step S47, and if only one potential transition 

15 clip exists, the process proceeds to step S48. 

[0090] In step S47, an optimal transition clip is 
selected from among the potential transition clips obtained 
in step S44. In this step, an optimal transition clip may 
be selected based on the importance, or the user may select 

20 a desired transition clip. The process in which the user 

selects a transition clip from among potential transition 
clips will be described later with reference to the 
flowchart shown in Fig. 6. 

[0091] In step S48, it is determined whether or not 
25 setting of a detailed item has been instructed for the 
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transition clip selected in step S47. If setting has been 
instructed, the process proceeds to step S49, and otherwise, 
the process proceeds to step S410. Setting of a detailed 
item is instructed by selecting the detailed setting button 
5 25 shown in Fig. 2, whereby the operator can arbitrarily set 

detailed information such as the direction and length of the 
transition clip. 

[0092] In step S49, a data processing system accepts 
setting of a detailed item by the user. The user operates 

10 the keyboard (KB) 16 so as to input detailed information 

regarding the transition clip. A screen displayed when 
detailed items are set and detailed items which can be set 
are different depending on the type of transition clip. 
[0093] In step S410, the transition clip determined in 

15 step S47 and the detailed information input in step S49 are 

stored in a table holding a video editing status . 
[0094] The edited data is processed by rendering based on 
the stored editing status, so that a final video file is 
automatically created based on video/audio files. 

20 [0095] Next, another method for obtaining potential 

transition clips in step S44 in Fig. 4 will be described 
with reference to Fig. 5. 

[0096] Fig. 5 is a flowchart showing step S44 in Fig. 4 
in detail. In this process, the metadata of the scenes 
25 obtained in step S43 is checked so as to obtain potential 
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transition clips suitable for the transition of the two 
scenes . 

[0097] In step S51, the metadata attached to the video 
data is analyzed so as to determine the relationship between 
5 the two scenes in the entire story and the feature of each 

scene. Fig. 10 shows an example of a template defining the 
correlation among event information, pieces of sub-event 
information included in the event information, and objects 
in the metadata, and features of the event information and 

10 objects. The metadata can be analyzed by referring to these 

pieces of information. For example, in Fig. 10, when the 
event information representing a first scene is E2 and when 
the event information representing a second scene is E3, the 
first and second scenes have an R2 relationship. The 

15 relationship between the two scenes is not limited to only 

one, but the two scenes may have a plurality of 
relationships . 

[0098] In step S52, meaning classification of transition 
clips suitable for transition of the two scenes is detected 

20 based on the analysis result of the metadata obtained in 

step S51. The table shown in Fig. 9 is stored in a storage 
device such as the DISK 14, the ROM 12 , the RAM 13, or the 
MemCard 15 in Fig. 1, and shows the relationship between 
events-objects relationship and information in which 

25 transition clips are classified by meaning based on the 
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impression and effect of each transition clip. By referring 
to this table, meaning classification of a transition clip 
corresponding to the relationship between the metadata 
attached to the two scenes is detected. For example, if 
5 relationship R2 has been obtained in the analysis in step 

S51, the meaning classification, such as emphasis, change, 
and induction, associated with R2 is detected. If the two 
scenes have a plurality of relationships , all meanings 
associated with the relationships are detected. 

10 [0099] In step S53, a potential transition clip is 

searched for based on the meaning classification detected in 
step S52. Fig. 8 shows a table showing meaning 
classification and other information corresponding to the 
title of each transition clip. A potential transition clip 

15 is searched for by referring to this table. If a plurality 

of meanings have been detected, all transition clips to 
which the meanings are attached are searched for, and the 
sum thereof is regarded as a search result . 

[0100] Next, a step of determining a transition clip in 
20 step S47 in Fig. 4 will be described with reference to Fig. 

6. 

[0101] Fig. 6 is a flowchart showing step S47 in Fig. 4 
in detail. In this process, the user determines a desired 
transition clip from among the potential transition clips 
25 extracted in step S44. 
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[0102] In step S61, various pieces of information about 
the potential transition clips extracted in the process 
shown in Fig. 4 are obtained so as to use them in the DISK 
14 and the RAM 13. 
5 [0103] In step S62, the potential transition clips 

extracted in the process shown in Fig. 4 are displayed to 
the user. The list of the potential transition clips are 
displayed in, for example, the CRT 20. Fig. 2 shows an 
example of the displayed list. In this example, a window 
10 system is used, and a transition clip is to be inserted 

between the scene of makeover and the scene of candle 
service in video data obtained by recording a wedding 
reception. 

[0104] In step S63, the data processing system accepts 
15 specification of a transition clip by the user. The user 

operates the keyboard (KB) 16 so as to specify a desired 
transition clip from among the potential transition clips 
displayed in step S62. 

[0105] The transition clips are expressed by using 
20 technical terms, and thus they are difficult to understand 

for beginners who don't have expert knowledge about video 
editing. Therefore, it is desirable to present information 
about each of potential transition clips to the user so that 
the user can easily select a transition clip. For example, 
25 an image of changing scenes may be expressed by animation. 
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or explanation of each transition clip may be displayed, 
[0106] Fig. 7 is an example of a table showing the 
relationship between event information of metadata attached 
to two scenes and transition clips. In step S44 in Fig. 4, 
5 by checking the metadata of the two scenes by using this 

table, potential transition clips suitable for transition of 
the two scenes can be extracted. For example. Fig. 7 shows 
that transition clips such as open heart, cross fade, and 
slide are suitable for transition from makeover to candle 
10 service, which are sub-event information included in event 

information of a reception. 

[0107] These pieces of information can be stored in the 
DISK 14 or the like shown in Fig. 1. In this embodiment, 
transition of scenes of home video contents can be 
15 appropriately performed by using a piece of event 

information as a unit. However, the method of the present 
invention can be applied to contents other than video 
contents by setting a reference unit depending on the type 
of contents. 

20 [0108] Fig. 8 is a table showing information used for 

searching for potential transition clips. In this table, 
various pieces of information are attached to the title of 
each transition clip. For example, in this embodiment, the 
table includes information about effects of transition clips, 

25 which are classified based on impression and meaning of each 
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transition clip, and intensity of impression and effects of 
each transition clip are indicated with values. 
[0109] The intensity is represented by absolute values 
from 0 to 10, and a positive/negative sign represents the 
5 application state of effects. That is, when the intensity 

is represented by a positive number, association in the 
meaning is stronger (impression is more intense) as the 
number is larger. On the other hand, when the intensity is 
represented by a negative number, association is weaker 

10 (opposite meaning becomes stronger) as the number is larger. 

For example, "fuzzy" corresponding to a transition clip 
"cross fade" gives ah impression (effect) to the user with 
an intensity of "10". On the other hand, "modulation" is 
represented by a negative number, which means it gives an 

15 opposite impression (effect) to the user with an intensity 

of "9". 

[0110] Also, files and text used for displaying images 
and explanations of transition clips in the areas 23 and 24 
in Fig. 2 are stored. 

20 [0111] These pieces of information and files are stored 

in a recording medium such as the hard disk (DISK) 14 in Fig. 
1. Further, the information and files may be stored in the 
PC on the LAN through the communication device (NCU) 22 or 
in a computer on the external network through the receiver 

25 (DTU) 23 in Fig. 1. 
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[0112] Fig. 9 shows an example of a table showing the 
relationship between events/objects in metadata and 
information obtained by classifying meanings of transition 
clips based on the impression and effect of each transition 
5 clip. By using such information, meaning classification 

suitable for transition of the two scenes can be detected 
based on the result of metadata analysis in step S52 in Fig. 
5. 

[0113] In Fig. 9, Rn (n is an integer) represents the 
10 relationship between pieces of event information En (n is an 

integer) or pieces of object information Objn (n is an 
integer) , and meaning classification of transition clip is 
associated with each relationship. 

[0114] For example, when two pieces of event information 
15 are associated with each other as "cause and effect" by 

relationship R2, the relationship between the two scenes can 
be impressed by a transition clip having meanings or effects 
of "emphasize the latter scene", "change", and "induction". 
[0115] These pieces of information can be stored in the 
20 DISK 14 or the like shown in Fig. 1. This embodiment is 

suitable for transition of scenes in video data or the like. 
However, the method of the present invention can be applied 
to data other than video data by selecting a transition 
effect corresponding to each type of data. 
25 [0116] Fig. 10 shows an example of a template defining 
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the correlation among event information in metadata, pieces 
of sub-event information included in the event information, 
and pieces of object information. By using these pieces of 
information, metadata is analyzed in step S51 in Fig. 5, so 
5 as to determine the relationship between two scenes in the 

entire story and the feature of each scene. 

[0117] In Fig. 10, En (n is an integer) represents event 
information and Objn (n is an integer) represents object 
information. A piece of event information includes a 

10 plurality of pieces of event information having time and 

cause-effect relationships. Also, each piece of event 
information includes object information about people and 
objects related to the event. Every piece of event 
information has a relationship with another, and also every 

15 piece of object information has a relationship with another. 

Each relationship can be represented by Rn (n is a number). 
Further, each piece of event information and object 
information can have various features. 
[0118] For example, in a wedding reception, event 

20 information El "wedding reception" is connected to sub-event 

information E2 "bride and groom in anteroom" and sub-event 
information E3 "entrance of bride and groom" by a 
relationship Rl. Also, the sub-event information E2 and E3 
of the event information El are connected by a relationship 

25 R2. Further, object information Objl "groom" and object 
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information Obj2 "bride" in the event information are 
connected by a romantic relationship R4 . 

[0119] These pieces of information can be stored in the 
DISK 14 or the like in Fig. 1. In this embodiment, home 
5 video contents can be appropriately analyzed by using a 

piece of event information or object information such as 
characters as a unit. However, the method of the present 
invention can be applied to contents other than video 
contents by setting a reference unit depending on the type 

10 of contents. 

[0120] In this way, the correlation among pieces of event 
information and object information and features of the 
information are defined in advance, and the information can 
be used for analyzing metadata. 

15 [0121] As is clear from the above description, according 

to this embodiment, the user can easily select a transition 
clip which is the most suitable for the relationship, 
contents, time, and place of arbitrary two scenes based on 
the impression and meaning of each transition clip. 

20 Accordingly, even if the user does not have expert knowledge 

about editing, he or she can easily edit video data. 
< Second Embodiment > 

[0122] In the first embodiment, suitable potential 
transition clips are extracted based on metadata of 
25 multimedia data so as to select a transition clip from among 
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them. Alternatively, unsuitable potential transition clips 
may be extracted based on metadata of multimedia data, and 
an error message may be generated when a user tries to 
select an unsuitable transition clip. 
5 [0123] Hereinafter, a process of editing video data by 

using a transition clip in an information processor 
according to a second embodiment will be described with 
reference to a specific example . 

[0124] Fig. 11 is a flowchart showing a process of 
10 inserting a transition clip when video data is edited. 

[0125] Steps S41 to S43 are the same as in the first 
embodiment, and thus the corresponding description is 
omitted. 

[0126] In step S114, the metadata of the two scenes 
15 obtained in step S43 is checked so as to extract transition 

clips which are unsuitable for transition of the two scenes . 
An unsuitable transition clips can be extracted by referring 
to the table as shown in Fig. 7, as in the first embodiment. 
That is, unsuitable transition clips can be extracted by 
20 using a table indicating transition clips which are 

unsuitable for events in the two scenes. 
[0127] Alternatively, the relationship between the 
metadata attached to the specified scenes may be analyzed, 
and an unsuitable transition clip may be searched for based 
2 5 on the analysis result and on the meaning and effects of 
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each transition clip. The process performed in that case 
will be described later with reference to a flowchart shown 
in Fig. 12. 

[0128] In step S115, the transition clip(s) obtained in 
5 step SI 14 is (are) stored in a recording medium such as the 

RAM 13. 

[0129] Steps S44 to S410 are the same as in the first 
embodiment, and thus the corresponding description is 
omitted. 

10 [0130] Fig. 12 is a flowchart showing step S114 in Fig. 

11 in detail. In this process, the metadata of the two 
scenes obtained in step S43 is analyzed and checked so as to 
extract a transition clip which is unsuitable for transition 
of the two scenes . 

15 [0131] In step S121, the metadata attached to the video 

data is analyzed so as to determine the relationship between 
the two scenes in the entire story and the feature of each 
scene. As in the first embodiment, the metadata is analyzed 
by referring to the information shown in Fig. 10. 

20 [0132] For example, in Fig. 10, when the event 

information representing a first scene is E2 and when the 
event information representing a second scene is E3, the 
first and second scenes have an R2 relationship. The 
relationship between the two scenes is not limited to only 

2 5 one, but the two scenes may have a plurality of 
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relationships . 

[0133] In step S122, meaning classification of transition 
clips suitable for transition of the two scenes is detected 
based on the analysis result of the metadata obtained in 
5 step S121. As in the first embodiment, by referring to the 

information shown in Fig. 9, meaning classification of a 
transition clip corresponding to the relationship between 
metadata attached to the two scenes is detected. For 
example, if relationship R2 has been obtained in analysis in 

10 step S121, meanings such as emphasis, change, and induction 

associated with R2 are detected. If there are a plurality 
of relationships between the two scenes, all the meanings 
associated with the relationships are detected. 
[0134] In step S123, an unsuitable transition clip is 

15 searched for based on the meaning classification detected in 

step S122. As in the first embodiment, an unsuitable 
transition clip can be searched for by referring to the 
table shown in Fig. 8. For example, in Fig. 8, a negative 
sign attached to the intensity of a transition clip 

20 indicates an opposite impression and meaning. Therefore, 

when an unsuitable transition clip is extracted in this 
embodiment, all transition clips having an intensity of 
negative number corresponding to the detected meaning 
classification are searched for, and the sum is regarded as 

25 a result. 
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[0135] Fig. 13 is an example of an error message 
displayed when the user specifies an unsuitable transition 
clip from among a plurality of transition clips. This is an 
example when a window system is used, and this message is 
5 displayed in the display device (CRT) 20 by the information 

processor of this embodiment. By displaying such a message, 
the information processor notifies the user that the 
specified transition clip is unsuitable for transition of 
the scenes. When the user presses the OK button, this 
10 screen disappears, so that the user can specify a desired 

clip again from among transition clips displayed in a list 
by using a transition clip selection screen. 
< Third Embodiment > 

[0136] In the first embodiment, suitable potential 
15 transition clips are extracted based on metadata of 

multimedia data, and then an optimal transition clip is 
determined. Alternatively, suitability of each transition 
clip (value indicating suitability of each transition clip 
for edited frames) may be calculated and displayed based on 
20 metadata of multimedia data, so that the user can determine 

a transition clip by referring to the suitability. 
Hereinafter, a process of editing video data by using a 
transition clip in an information processor according to a 
third embodiment will be described with reference to a 
25 specific example. 
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[0137] Fig. 14 is an example of a screen displayed when 
the user specifies a desired clip from among a plurality of 
transition clips in the process shown in Fig. 6. This in an 
example using a window system, and this screen is displayed 
5 in the display device (CRT) 20 by the information processor 

of this embodiment . 

[0138] In Fig. 14, reference numerals 21 and 23 to 28 are 
the same as those in the first embodiment shown in Fig. 2, 
and thus the corresponding description will be omitted. 

10 [0139] Reference numeral 142 denotes a list box. 

Transition clips suitable for transition of the scenes 
specified by an operator are displayed in this list box, so 
that the operator can specify a transition clip to be 
inserted. In the list box, the suitability of each 

15 transition clip is shown at the right side, and the user can 

check how much each transition clip is suitable for 
transition of the specified scenes. 

[0140] In this embodiment, suitability is represented by 
decimal from 0 to 1, and the suitability is higher as the 

20 number is closer to 1. Also, all the transition clips found 

by search need not be displayed in the list box, but only 
transition clips having a predetermined suitability or top- 
10 transition clips in suitability ranking may be displayed. 
The transition clips are sorted in the descending ranking of 

25 suitability. In Fig. 14, the suitability of "open heart" is 
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0.85, that of "cross zoom" is 0.78, and that of "slid in" is 
0.75. "Cross zoom" is currently selected and is highlighted. 
When the operator presses a cursor -movement key on the 
keyboard (KB) 16, the highlight is shifted from "cross zoom" 
to "open heart" or "slide in". In this way, the operator 
can arbitrarily specify a desired transition clip in the 
list. 

[0141] As in the first embodiment, metadata attached to 
video data is used for setting a transition effect in this 
embodiment. The metadata can be described in accordance 
with a method standardized by, for example, MPEG- 7. 
[0142] Next, a process of editing video data by using a 
transition clip in the information processor according to 
this embodiment will be described with reference to a 
specific example. 

[0143] Fig. 15 is a flowchart showing a process of 
inserting a transition clip when video data is edited. 
[0144] Steps S41 to S43 are the same as in the flowchart 
shown in Fig. 4 of the first embodiment, and thus the 
corresponding description will be omitted. 
[0145] In step S154, the metadata of the two scenes 
obtained in step S43 is checked so as to search for a 
potential transition clip which is suitable for transition 
of the two scenes. When a potential transition clips is 
searched for, a suitable transition clip can be extracted by 
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analyzing the relationship between metadata attached to the 
two scenes and calculating the suitability of each 
transition clip based on the analysis result, meaning and 
effect of each transition clip, and the importance. A 
5 process performed in that case will be described later with 

reference to the flowchart shown in Fig. 16. 
[0146] In step S155, it is determined whether or not a 
plurality of transition clips have been obtained in step 
S154. If a plurality of transition clips have been obtained, 
10 the process proceeds to step S156, and if only one 

transition clip has been obtained, the process proceeds to 
step S48. 

[0147] In step S156, an optimal transition clip is 
determined from among the transition clips obtained in step 

15 S154. A transition clip having the highest suitability may 

be selected in accordance with the suitability found in step 
S154. Alternatively, some transition clips having a 
predetermined suitability or some top-ranked transition 
clips may be presented to the user based on the result 

20 obtained in step S154, so that the user is allowed to select 

a desired transition clip from among these transition clips. 
A process of specifying a transition clip by the user is the 
same as that shown in Fig. 6 of the first embodiment, and 
thus the corresponding description will be omitted. Also, 

25 steps S48 to S410 are the same as in the flowchart shown in 



- 38 - 



Fig. 4 of the first embodiment, and thus the corresponding 
description will be omitted. 

[0148] Fig. 16 is a flowchart showing step S154 in Fig. 
15 in detail. In this process, an optimal transition clip 
5 is determined by calculating the suitability of each 

transition clip based on importance. 

[0149] In step S161, the metadata of the two scenes 
obtained in step S43 in Fig. 15 is checked so as to extract 
potential transition clips suitable for transition of the 

10 two scenes. For example, the relationship between the 

metadata attached to the two scenes is analyzed, and then 
suitable transition clips can be searched for based on the 
analysis result and the meaning and effect of a transition 
clip. A process performed in that case will be described 

15 later with reference to the flowchart shown in Fig. 17. 

[0150] In step S162, the intensity value corresponding to 
meaning classification detected in step S172 in Fig. 17 of 
each of the transition clips extracted in step S161 is 
obtained by referring to the table shown in Fig. 8. A 

20 plurality of meanings may be detected in step S172, and a 

plurality of meanings may be associated with a transition 
clip. Therefore, intensity values for all the meanings 
detected in step S172 are obtained. The obtained intensity 
values are stored in a work memory (not shown) in the RAM 13. 

25 [0151] Then, in step S163, the suitability of each 
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transition clip is calculated. First, the sum of the 
intensity values stored in the RAM 13 is obtained as 
suitability, and the suitability is stored in a region in 
the RAM 13 corresponding to each transition clip. 
5 [0152] This process is performed for all the transition 

clips obtained in step S161. Then, in step S164, the values 
of suitability calculated for the transition clips are 
sorted in descending order. 

[0153] A step of determining a transition clip in step 
10 S156 in Fig. 15 is the same as in the flowchart shown in Fig. 

6 of the first embodiment, and thus the corresponding 
description will be omitted. 

[0154] Next, a step of extracting potential transition 
clips in step S161 in Fig. 16 will be described in detail 

15 with reference to Fig. 17. 

[0155] Fig. 17 is a flowchart showing step S161 in Fig. 
16 in detail. In this process, potential transition clips 
suitable for transition of the two scenes are extracted by 
checking the metadata of the two scenes obtained in step S43 

20 in Fig. 15. 

[0156] In step S171, the metadata attached to the video 
data is analyzed so as to determine the relationship between 
the two scenes in the entire story and the feature of each 
scene. As in the first embodiment, the metadata is analyzed 

25 by referring to the information shown in Fig. 10. For 
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example, in Fig. 10, the two scenes are connected by a 
relationship R2 . The relationship between the two scenes is 
not always only one, but the two scenes may have a plurality 
of relationships . 
5 [0157] In step S172, meaning classification of transition 

clips suitable for transition of the two scenes is detected 
based on the analysis result of the metadata obtained in 
step S171. As in the first embodiment, by referring to the 
information shown in Fig. 9, meaning classification of a 

10 transition clip corresponding to the relationship between 

metadata attached to the two scenes is detected. 
[0158] For example, if relationship R2 has been obtained 
in analysis in step S171, meanings such as emphasis, change, 
and induction associated with R2 are detected. If there are 

15 a plurality of relationships between the two scenes, all the 

meanings associated with the relationships are detected. 
[0159] In step S173, a potential transition clip is 
searched for based on the meaning classification detected in 
step S172. A potential transition clip is searched for by 

20 referring to the table shown in Fig. 8, as in the first 

embodiment. If a plurality of meanings are detected, 
transition clips corresponding to all the meanings are 
searched for, and the sum thereof is regarded as a search 
result . 

2 5 [0160] As is clear from the above description, according 
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to this embodiment, the user can easily understand the 
feature of each transition clip by indicating a suitability 
value, and thus the user can select a transition clip easily, 
<Other Embodiment s> 
5 [0161] In the above -described embodiments, video data is 

used as accumulated information to be edited. However, the 
present invention can be applied to multimedia data other 
than video data, such as image data and audio data, by 
setting attached metadata, method for analyzing metadata, 
10 and a transition effect corresponding to the type of 

contents . 

[0162] Also, in the above-described embodiments, suitable 
transition clips are extracted by analyzing the metadata 
shown in Fig, 3, that is, information indicating the 

15 contents of video data including keywords of event 

information, characters, situation, and place, by using the 
template showing the correlation between event information 
and object information of metadata shown in Fig. 10. 
Alternatively, transition clips can be extracted by 

20 attaching metadata describing the relationship between event 

information and objects to video data so as to use the 
relationship between metadata and meaning classification of 
transition clips shown in Fig. 9. 

[0163] Also, transition clips can be extracted by 
25 attaching metadata describing the relationship between 
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scenes to video data so as to define the relationship 
between the scenes and the relationship between transition 
clips (not shown). 

[0164] Further, in the above-described embodiments, video 
5 data input to a computer device is edited so as to set a 

transition effect between scenes. However, the present 
invention may be realized as a part of video editing 
function provided in a recording device such as a video 
camera, so that a transition effect can be added during or 

10 after video recording. In that case, the metadata shown in 

Fig. 3, the information defining the correlation and 
features of event information and object information shown 
in Fig. 9, the information attached to a transition clip 
shown in Fig. 10, and so on must be stored in a storage 

15 device such as a DISK, ROM, RAM, or memory card of the 

recording device. These pieces of information can be 
obtained through a LAN or the like and stored in the storage 
device. The video data edited during recording is processed 
by rendering and is stored in the storage device of the 

20 video camera. 

[0165] In the above-described embodiments, a transition 
effect is set between scenes so as to edit video data. 
However, the present invention can be applied when the video 
data is not edited or processed but a plurality of scenes 

25 are sequentially played back. In that case, too, a suitable 
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transition effect can be inserted between scenes . 
[0166] The present invention may be applied to a system 
including a plurality of devices (for example, host computer, 
interface device, reader, and printer) or may be applied to 
5 a single device (for example, copier or fax machine). 

[0167] Of course, the object of the present invention can 
be achieved by supplying a storage medium (or recording 
medium) containing software program codes for realizing the 
function of the above-described embodiments to a system or 

10 apparatus so that a computer (or CPU or MPU) of the system 

or apparatus reads and executes the program codes stored in 
the storage medium. In this case, the program codes read 
from the storage medium realize the function of the above- 
described embodiments, and thus the storage medium storing 

15 the program codes is included in the present invention. As 

the storage medium for supplying the program codes , a floppy 
(registered trademark) disk, a hard disk, an optical disk, a 
magnetooptical disk, a CD-ROM, a CD-R, a magnetic tape, a 
nonvolatile memory card, or a ROM may be used. 

20 [0168] The function of the above-described embodiments 

can be realized when a computer executes read program codes 
or when an operating system (OS) operated in the computer 
executes part or whole of actual processing based on the 
instructions of the program codes . 

25 [0169] Further, the function of the above -described 
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embodiments may be realized when the program codes read from 
the storage medium is written into a memory provided in an 
expansion board inserted into the computer or an expansion 
unit connected to the computer and then a CPU or the like 
5 provided in the expansion board or expansion unit executes 

part or whole of the processing based on the instructions of 
the program codes . 

[0170] As described above, according to the present 
invention, even if a user does not have expert knowledge 
10 about data editing, he or she can edit video data by 



inserting a transition clip between scenes. Accordingly, 



even a user who is not familiar with data editing can create 



sophisticated video having a video effect . 



[0171] 



While the present invention has been described 



15 



with reference to what are presently considered to be the 



preferred embodiments, it is to be understood that the 



invention is not limited to the disclosed embodiments . 



On 



the contrary, the invention is intended to cover various 



modifications and equivalent arrangements included within 



20 



the spirit and scope of the appended claims . The scope of 



the following claims is to be accorded the broadest 



interpretation so as to encompass all such modifications and 



equivalent structures and functions. 



